Mobile terminal and control method therefor

ABSTRACT

A mobile terminal according to the present disclosure includes: GPS configured to receive location information of the mobile terminal, a wireless communication unit configured to perform wireless communication with an external device, and a controller configured to store communication information when a communication event with the external device is generated. The controller matches the location information received from the GPS with the communication information and stores the matched information when the communication event is generated, and outputs at least one information corresponding to a search query for retrieving communication information from pre-stored communication information by using location information matched to the pre-stored communication information and the search query, when the search query is input by a user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is the National Phase of PCT International ApplicationNo. PCT/KR2019/006707 filed on Jun. 4, 2019, which claims the prioritybenefit under 35 U.S.C. § 119(e) to U.S. Provisional Application No.62/808,255 filed on Feb. 20, 2019, all of which are hereby expresslyincorporated by reference into the present application.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to a mobile terminal having an artificialintelligence unit, and a control method therefor.

Discussion of the Related Art

Artificial intelligence (AI) is an area of computer engineering andinformation technology (IT) that aims to create computers capable ofperforming tasks requiring human intelligence such as thinking,learning, and self-development, which enables the computers to mimichuman intellectual behavior.

In addition, artificial intelligence does not just exist in itself. Itis directly and indirectly related to other areas of computer science inmany ways. Nowadays, there have been active attempts to use artificialintelligence elements in a wide range of fields of informationtechnology, especially in problem solving.

Meanwhile, in the related art, a context awareness technology usingartificial intelligence has been actively studied for recognizing auser's current situation and providing necessary information to a userin a desirable manner.

With the development of the context awareness technology, there is anincreasing demand for a mobile terminal capable of properly respondingaccording to the user's current situation.

SUMMARY OF THE INVENTION

One aspect of the present disclosure is directed to solving theabove-mentioned problems and other drawbacks. Another aspect of thepresent disclosure is directed to searching pre-stored communicationinformation using a location-related keyword.

Still another aspect of the present disclosure is directed to searchingpre-stored communication information using a location-related keywordand a time-related keyword together.

In order to achieve the aspects and other advantages, there is provideda mobile terminal including GPS configured to receive locationinformation of the mobile terminal, a wireless communication unitconfigured to perform wireless communication with an external device, acontroller configured to store communication information when acommunication event with the external device is generated. Thecontroller may match the location information received from the GPS withthe communication information and stores the matched information whenthe communication event is generated, and output at least oneinformation corresponding to a search query for retrieving communicationinformation from pre-stored communication information by using locationinformation matched to the pre-stored communication information and thesearch query, when the search query is input by a user.

In one embodiment, the controller may extract a keyword related tolocation information from the search query, compares the extractedkeyword with location information matched to pre-stored communicationinformation, and outputs at least one piece of information correspondingto the search query from the pre-stored communication information.

In one embodiment, when location information matched to thecommunication information satisfies a preset condition, the controllermay store the location information by matching the location informationwith a representative name assigned to the location information.

In one embodiment, the controller may change the extracted keyword tolocation information corresponding to any one representative name ofpre-stored representative names when the extracted keyword is identicalto the any one representative name, compares the changed locationinformation with location information matched to pre-storedcommunication information, and outputs at least one piece of informationcorresponding to the search query from the pre-stored communicationinformation.

In one embodiment, the controller may determine that the presetcondition is satisfied when the same location information is collectedmore than a predetermined number of times within a predetermined timeperiod, and store the location information by matching the locationinformation with the representative name.

In one embodiment, the controller may extract location information fromthe communication information, and store the extracted locationinformation by matching the extracted location information with thecommunication information.

In one embodiment, at least part of the pre-stored communicationinformation may be matched with a plurality of pieces of locationinformation.

In one embodiment, the controller may a keyword related to locationinformation from the search query, and output communication informationcorresponding to the plurality of pieces of location information whenthe extracted keyword is identical to any one of the plurality of piecesof location information.

In one embodiment, the controller may extract a keyword related tolocation information and a keyword related to time information from thesearch query, and output information corresponding to each of theextracted keywords from the pre-stored communication information.

In one embodiment, the controller may compare the extracted keywordrelated to location information with location information matched topre-stored communication information to extract some of the pre-storedcommunication information, and output information corresponding to theextracted keyword related to time information from the extractedcommunication information.

In one embodiment, the controller may determine that a search query isinput from the user when a preset condition is satisfied, and uselocation information matched to pre-stored communication information andthe search query to output at least one piece of informationcorresponding to the search query from the pre-stored communicationinformation.

In one embodiment, the preset condition may be that executing a specificapplication.

In one embodiment, the controller may collect location information fromthe GPS, compare the collected location information with locationinformation matched to pre-stored communication information, and outputat least one piece of information corresponding to the collectedlocation information from the pre-stored communication information, whenthe preset condition is satisfied.

In one embodiment, the controller may collect current time information,and output at least one piece of information corresponding to thecollected location information from pre-stored communication informationwhen the preset condition is satisfied.

In order to achieve the aspects and other advantages, there is alsoprovided a method for controlling a mobile terminal, the methodincluding storing communication information when a communication eventwith an external device is generated, collecting location information ofthe mobile terminal from GPS when the communication event is generated,matching the location information received from the GPS with thecommunication information to store the matched information, andoutputting at least one information corresponding to a search query forretrieving communication information from pre-stored communicationinformation by using location information matched to the pre-storedcommunication information and the search query, when the search query isinput by a user.

Advantageous Effects

According to the present disclosure, it is possible to retrieve callhistory or a list of sent and received text messages of a user by usinga location (place) that the user can intuitively think of and a time asa search query.

In addition, according to the present disclosure. location informationcollected when a communication event is generated is stored togetherwith communication information, thereby facilitating later use of thestored communication information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a mobile terminal according to anembodiment of the present disclosure.

FIGS. 1B and 1C are conceptual views illustrating one example of amobile terminal according to the present disclosure, viewed fromdifferent directions.

FIG. 2 is a conceptual view illustrating a method in which information(data) is stored by matching communication information and communicationevent related information with another information.

FIG. 3 is a block diagram illustrating searching for communicationinformation and event related information in a mobile terminal accordingto one embodiment of the present disclosure.

FIG. 4 is a flowchart illustrating one embodiment of searchingpre-stored data using time information included in a search query.

FIGS. 5 and 6 are conceptual views illustrating searching pre-storeddata using location information or time information included in a searchquery, in accordance with one embodiment of the present disclosure.

FIG. 7 is a conceptual view illustrating one embodiment of extracting alocation-related keyword included in a search query in the mobileterminal according to the present disclosure.

FIG. 8 is a conceptual view illustrating one embodiment of retrieving animage previously captured by using time and location information.

FIG. 9 is a conceptual view illustrating one embodiment in whichspecific information is provided to a user by retrieving the specificinformation based on a terminal usage pattern of the user even whenthere is no search request of the user.

FIG. 10 is a flowchart illustrating one embodiment of providing areminder to a user using a pre-stored text message.

FIGS. 11A and 11B are conceptual views illustrating providing a reminderto a user using a pre-stored text message, in accordance with oneembodiment of the present disclosure.

FIGS. 12A and 12B are conceptual views illustrating a state in which twoor more artificial intelligence (AI) agents are interconnected.

FIG. 13 is a conceptual view illustrating one embodiment in which asearch command is input via an AI agent installed in an image displaydevice, and a search is performed in a mobile terminal.

FIG. 14 is a conceptual view illustrating one embodiment in which asearch command is input via an AI agent installed in a mobile terminal,and a search result is displayed on an image display device.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Description will now be given in detail according to exemplaryembodiments disclosed herein, with reference to the accompanyingdrawings. For the sake of brief description with reference to thedrawings, the same or equivalent components may be provided with thesame or similar reference numbers, and description thereof will not berepeated. In general, a suffix such as “module” and “unit” may be usedto refer to elements or components. Use of such a suffix herein ismerely intended to facilitate description of the specification, and thesuffix itself is not intended to give any special meaning or function.In describing the present disclosure, if a detailed explanation for arelated known function or construction is considered to unnecessarilydivert the gist of the present disclosure, such explanation has beenomitted but would be understood by those skilled in the art. Theaccompanying drawings are used to help easily understand the technicalidea of the present disclosure and it should be understood that the ideaof the present disclosure is not limited by the accompanying drawings.The idea of the present disclosure should be construed to extend to anyalterations, equivalents and substitutes besides the accompanyingdrawings.

It will be understood that although the terms first, second, etc. may beused herein to describe various elements, these elements should not belimited by these terms. These terms are generally only used todistinguish one element from another.

It will be understood that when an element is referred to as being“connected with” another element, the element can be connected with theanother element or intervening elements may also be present. Incontrast, when an element is referred to as being “directly connectedwith” another element, there are no intervening elements present.

A singular representation may include a plural representation unless itrepresents a definitely different meaning from the context.

Terms such as “include” or “has” are used herein and should beunderstood that they are intended to indicate an existence of severalcomponents, functions or steps, disclosed in the specification, and itis also understood that greater or fewer components, functions, or stepsmay likewise be utilized.

Mobile terminals presented herein may be implemented using a variety ofdifferent types of terminals. Examples of such terminals includecellular phones, smart phones, user equipment, laptop computers, digitalbroadcast terminals, personal digital assistants (PDAs), portablemultimedia players (PMPs), navigators, portable computers (PCs), slatePCs, tablet PCs, ultra books, wearable devices (for example, smartwatches, smart glasses, head mounted displays (HMDs)), and the like.

By way of non-limiting example only, further description will be madewith reference to particular types of mobile terminals. However, suchteachings apply equally to other types of terminals, such as those typesnoted above. In addition, these teachings may also be applied tostationary terminals such as digital TV, desktop computers, and thelike.

Referring to FIGS. 1A to 1C, FIG. 1A is a block diagram of a mobileterminal according to an embodiment of the present disclosure, and FIGS.1B and 1C are conceptual views illustrating one example of a mobileterminal, viewed from different directions.

The mobile terminal 100 may be shown having components such as awireless communication unit 110, an input unit 120, an artificialintelligence unit (or system) 130, a sensing unit 140, an output unit150, an interface unit 160, a memory 170, a controller 180, and a powersupply unit 190. It is understood that implementing all of theillustrated components is not a requirement, and that greater or fewercomponents may alternatively be implemented.

In more detail, the wireless communication unit 110 may typicallyinclude one or more modules which permit communications such as wirelesscommunications between the mobile terminal 100 and a wirelesscommunication system, communications between the mobile terminal 100 andanother mobile terminal, or communications between the mobile terminal100 and an external server. Further, the wireless communication unit 110may typically include one or more modules which connect the mobileterminal 100 to one or more networks.

The wireless communication unit 110 may include one or more of abroadcast receiving module 111, a mobile communication module 112, awireless Internet module 113, a short-range communication module 114,and a location information module 115.

The input unit 120 may include a camera 121 or an image input unit forobtaining images or video, a microphone 122, which is one type of audioinput device for inputting an audio signal, and a user input unit 123(for example, a touch key, a mechanical key, and the like) for allowinga user to input information. Data (for example, audio, video, image, andthe like) may be obtained by the input unit 120 and may be analyzed andprocessed according to user commands.

The artificial intelligence unit 130 performs processing of informationbased on an artificial intelligence (AI) technology, which may includeone or more modules that perform at least one of learning ofinformation, inference of information, perception of information, andnatural language processing.

The artificial intelligence unit 130 may use a machine learningtechnology to perform at least one of learning, inferring and processinga large amount of information (big data), such as information stored inthe mobile terminal, environmental information around the mobileterminal, information stored in an external storage capable ofperforming communication, and the like. The artificial intelligence unit130 may control the mobile terminal to predict (or infer) at least oneexecutable operation and execute an operation having the highestfeasibility among the predicted at least one operation, by using theinformation learned through the machine learning technology.

The machine learning technology is a technology that collects and learnsa large amount of information based on at least one algorithm, anddetermines (or evaluates) and predicts information based on the learnedinformation. The learning of information is an operation that figuresout characteristics, rules, and determination criteria of information,quantifies the relationship (correlation) between information andinformation, and predicts new data using a quantified pattern.

The at least one algorithm used by the machine learning technology maybe a statistical-based algorithm, for example, a decision tree that usesa tree structure type as a prediction model, an Artificial NeuralNetwork (ANN) that mimics neural network architecture and functions,genetic programming based on biological evolutionary algorithms,clustering that distributes observed examples into subsets of clusters,Monte Carlo method that computes function values through randomlyextracted random numbers from probability, and the like.

Deep learning, a subfield of machine learning, is a technology thatperforms at least one of learning, determining, and processing ofinformation using an ANN algorithm. The ANN may have an architecture inwhich layers are connected to transfer data between layers. This deeplearning technology may allow learning of a large amount of informationthrough the ANN using a graphic processing unit (GPU) optimized forparallel computing.

Meanwhile, the artificial intelligence unit 130 may collect (detect,monitor, extract, receive, etc.) signals, data, and information input oroutput to or from components of the mobile terminal in order to collectan enormous amount of information for applying the machine learningtechnology. The artificial intelligence unit 130 may also collect(sense, monitor, extract, detect, and receive) data, information, andthe like stored in the external storage (e.g., a cloud server). Morespecifically, collection of information may be understood as a term thatincludes operations of sensing information through a sensor, extractinginformation stored in the memory 170, or receiving information from theexternal storage through communications.

The artificial intelligence unit 130 may sense information in the mobileterminal, environmental information around the mobile terminal, and userinformation through the sensing unit 140. In addition, the artificialintelligence unit 130 may receive broadcast signals and/orbroadcast-related information, wireless signals, wireless data, and thelike through the wireless communication unit 110. Furthermore, theartificial intelligence unit 130 may receive image information (orsignal), audio information (or signal), data, or information input by auser from the input unit 120.

The artificial intelligence unit 130 collects a large amount ofinformation in the background in real time, learns the collectedinformation, and stores properly processed information (e.g., knowledgegraph, command policy, personalization database, communication engine,etc.) in the memory 170.

When an operation of the mobile terminal is predicted based on thelearned information using the machine learning technology, theartificial intelligence unit 130 may control the components of themobile terminal, or may transmit a control command for executing theoperation to the controller 180. Then the controller 180 may execute thepredicted operation by controlling the mobile terminal based on thecontrol command.

Meanwhile, when a specific operation is performed, the artificialintelligence unit 130 analyzes history information (or data) indicatingperformance of the specific operation through the machine learningtechnology, and updates the existing learned information based on theanalyzed information. Accordingly, accuracy of information predictionfor the artificial intelligence unit 130 can be improved.

Meanwhile, in this specification, the artificial intelligence unit 130and the controller 180 may be understood as the same component. In thiscase, functions performed by the controller 180 described herein may beexpressed as being performed by the artificial intelligence unit 130,and the controller 180 may be named as the artificial intelligence unit130, or conversely, the artificial intelligence unit 130 may be referredto as the controller 180.

Alternatively, in this specification, the artificial intelligence unit130 and the controller 180 may be understood as separate components. Inthis case, the artificial intelligence unit 130 and the controller 180may perform various controls on the mobile terminal 100 through mutualdata exchange. The controller 180 may perform at least one function onthe mobile terminal or may control at least one of the components of themobile terminal 100 based on a result derived from the artificialintelligence unit 130. Further, the artificial intelligence unit 130 mayalso operate according to the control of the controller 180.

The sensing unit 140 may typically be implemented using one or moresensors configured to sense internal information of the mobile terminal,the surrounding environment of the mobile terminal, user information,and the like. For example, the sensing unit 140 may include at least oneof a proximity sensor 141, an illumination sensor 142, a touch sensor,an acceleration sensor, a magnetic sensor, a G-sensor, a gyroscopesensor, a motion sensor, an RGB sensor, an infrared (IR) sensor, afinger scan sensor, a ultrasonic sensor, an optical sensor (for example,camera 121), a microphone 122, a battery gauge, an environment sensor(for example, a barometer, a hygrometer, a thermometer, a radiationdetection sensor, a thermal sensor, and a gas sensor, among others), anda chemical sensor (for example, an electronic nose, a health caresensor, a biometric sensor, and the like). The mobile terminal disclosedherein may be configured to utilize information obtained from one ormore sensors of the sensing unit 140, and combinations thereof.

The output unit 150 may typically be configured to output various typesof information, such as audio, video, tactile output, and the like. Theoutput unit 150 may be shown having at least one of a display unit 151,an audio output module 152, a haptic module 153, and an optical outputmodule 154. The display unit 151 may have an inter-layered structure oran integrated structure with a touch sensor in order to implement atouch screen. The touch screen may function as the user input unit 123which provides an input interface between the mobile terminal 100 andthe user and simultaneously provide an output interface between themobile terminal 100 and a user.

The interface unit 160 serves as an interface with various types ofexternal devices that are coupled to the mobile terminal 100. Theinterface unit 160, for example, may include any of wired or wirelessports, external power supply ports, wired or wireless data ports, memorycard ports, ports for connecting a device having an identificationmodule, audio input/output (I/O) ports, video I/O ports, earphone ports,and the like. In some cases, the mobile terminal 100 may performassorted control functions associated with a connected external device,in response to the external device being connected to the interface unit160.

The memory 170 is implemented to store data to support various functionsor features of the mobile terminal 100. The memory 170 may store aplurality of application programs or applications operated in the mobileterminal 100, data for operations of the mobile terminal 100, commands,data for operations of the artificial intelligence unit 130 (e.g., atleast one algorithm information for machine learning). Some of theseapplication programs may be downloaded from an external server viawireless communications. Also, at least some of these applicationprograms may be installed within the mobile terminal 100 at time ofmanufacturing or shipping, which is typically the case for basicfunctions of the mobile terminal 100 (for example, receiving a call,placing a call, receiving a message, sending a message, and the like).Application programs may be stored in the memory 170, installed in themobile terminal 100, and executed by the controller 180 to perform anoperation (or function) for the mobile terminal 100.

The controller 180 typically functions to control an overall operationof the mobile terminal 100, in addition to the operations associatedwith the application programs. The controller 180 may provide or processinformation or functions appropriate for a user by processing signals,data, information and the like, which are input or output by theaforementioned various components, or activating application programsstored in the memory 170.

Also, the controller 180 may control at least some of the componentsillustrated in FIG. 1A, to execute an application program that have beenstored in the memory 170. In addition, the controller 180 may control atleast two of those components included in the mobile terminal 100 toactivate the application program.

The power supply unit 190 may be configured to receive external power orprovide internal power in order to supply appropriate power required foroperating elements and components included in the mobile terminal 100.The power supply unit 190 may include a battery, and the battery may beconfigured to be embedded in the terminal body, or configured to bedetachable from the terminal body.

At least part of the components may cooperatively operate to implementan operation, a control or a control method of a mobile terminalaccording to various embodiments disclosed herein. Also, the operation,the control or the control method of the mobile terminal may beimplemented on the mobile terminal by an activation of at least oneapplication program stored in the memory 170.

Hereinafter, description will be given in more detail of theaforementioned components with reference to FIG. 1A, prior to describingvarious embodiments implemented through the mobile terminal 100.

First, regarding the wireless communication unit 110, the broadcastreceiving module 111 is typically configured to receive a broadcastsignal and/or broadcast associated information from an externalbroadcast managing entity via a broadcast channel. The broadcast channelmay include a satellite channel, a terrestrial channel, or both. In someembodiments, two or more broadcast receiving modules may be utilized tofacilitate simultaneous reception of two or more broadcast channels, orto support switching among broadcast channels.

The mobile communication module 112 can transmit and/or receive wirelesssignals to and from one or more network entities. Typical examples of anetwork entity include a base station. an external mobile terminal, aserver, and the like. Such network entities form part of a mobilecommunication network, which is constructed according to technicalstandards or communication methods for mobile communications (forexample, Global System for Mobile Communication (GSM), Code DivisionMulti Access (CDMA), CDMA2000 (Code Division Multi Access 2000), EV-DO(Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WidebandCDMA (WCDMA), High Speed Downlink Packet access (HSDPA), HSUPA (HighSpeed Uplink Packet Access), Long Term Evolution (LTE), LTE-A (Long TermEvolution-Advanced), and the like).

The wireless signal may include various types of data depending on avoice call signal, a video call signal, or a text/multimedia messagetransmission/reception.

The wireless Internet module 113 refers to a module for wirelessInternet access. This module may be internally or externally coupled tothe mobile terminal 100. The wireless Internet module 113 may transmitand/or receive wireless signals via communication networks according towireless Internet technologies.

Examples of such wireless Internet access include Wireless LAN (WLAN),Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Digital Living Network Alliance(DLNA), Wireless Broadband (WiBro), Worldwide Interoperability forMicrowave Access (WiMAX), High Speed Downlink Packet Access (HSDPA),High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE),LTE-advanced (LTE-A) and the like. The wireless Internet module 113 maytransmit/receive data according to one or more of such wireless Internettechnologies, and other Internet technologies as well.

When the wireless Internet access is implemented according to, forexample, WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A and the like,as part of a mobile communication network, the wireless Internet module113 performs such wireless Internet access. As such, the Internet module113 may cooperate with, or function as, the mobile communication module112.

The short-range communication module 114 is configured to facilitateshort-range communications. Suitable technologies for implementing suchshort-range communications include BLUETOOTH™, Radio FrequencyIDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand(UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity(Wi-Fi), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), andthe like. The short-range communication module 114 in general supportswireless communications between the mobile terminal 100 and a wirelesscommunication system, communications between the mobile terminal 100 andanother mobile terminal, or communications between the mobile terminal100 and a network where another mobile terminal (or an external server)is located, via wireless area networks. One example of the wireless areanetworks is a wireless personal area networks.

Here, another mobile terminal (which may be configured similarly tomobile terminal 100) may be a wearable device, for example, a smartwatch, a smart glass or a head mounted display (HMD), which is able toexchange data with the mobile terminal 100 (or otherwise cooperate withthe mobile terminal 100). The short-range communication module 114 maysense or recognize the wearable device, and permit communication betweenthe wearable device and the mobile terminal 100. In addition, when thesensed wearable device is a device which is authenticated to communicatewith the mobile terminal 100, the controller 180, for example, may causetransmission of at least part of data processed in the mobile terminal100 to the wearable device via the short-range communication module 114.Hence, a user of the wearable device may use the data processed in themobile terminal 100 on the wearable device. For example, when a call isreceived in the mobile terminal 100, the user may answer the call usingthe wearable device. Also, when a message is received in the mobileterminal 100, the user can check the received message using the wearabledevice.

The location information module 115 is generally configured to detect,calculate, derive or otherwise identify a position (or current position)of the mobile terminal. As an example, the location information module115 includes a Global Position System (GPS) module, a Wi-Fi module, orboth. For example, when the mobile terminal uses a GPS module, aposition of the mobile terminal may be acquired using a signal sent froma GPS satellite. As another example, when the mobile terminal uses theWi-Fi module, a position of the mobile terminal can be acquired based oninformation related to a wireless access point (AP) which transmits orreceives a wireless signal to or from the Wi-Fi module. If desired, thelocation information module 115 may alternatively or additionallyfunction with any of the other modules of the wireless communicationunit 110 to obtain data related to the position of the mobile terminal.The location information module 115 is a module used for acquiring theposition (or the current position) and may not be limited to a modulefor directly calculating or acquiring the position of the mobileterminal.

Examples of such inputs include audio, image, video, data, and userinput. Image and video input are often obtained using one or morecameras 121. Such cameras 121 may process image frames of still picturesor video obtained by image sensors in a video or image capture mode. Theprocessed image frames can be displayed on the display unit 151 orstored in memory 170. Meanwhile, the cameras 121 may be arranged in amatrix configuration to permit a plurality of images having variousangles or focal points to be input to the mobile terminal 100. Also, thecameras 121 may be located in a stereoscopic arrangement to acquire leftand right images for implementing a stereoscopic image.

The microphone 122 processes an external audio signal into electricaudio (sound) data. The processed audio data can be processed in variousmanners according to a function being executed in the mobile terminal100. If desired, the microphone 122 may include assorted noise removingalgorithms to remove unwanted noise generated in the course of receivingthe external audio signal.

The user input unit 123 is a component that permits input by a user.Such user input may enable the controller 180 to control operation ofthe mobile terminal 100. The user input unit 123 may include one or moreof a mechanical input element (for example, a mechanical key, a buttonlocated on a front and/or rear surface or a side surface of the mobileterminal 100, a dome switch, a jog wheel, a jog switch, and the like),or a touch-sensitive input element, among others. As one example, thetouch-sensitive input element may be a virtual key, a soft key or avisual key, which is displayed on a touch screen through softwareprocessing, or a touch key which is located on the mobile terminal at alocation that is other than the touch screen. On the other hand, thevirtual key or the visual key may be displayed on the touch screen invarious shapes, for example, graphic, text, icon, video, or acombination thereof.

The sensing unit 140 is generally configured to sense one or more ofinternal information of the mobile terminal, surrounding environmentinformation of the mobile terminal, user information, or the like, andgenerate a corresponding sensing signal. The controller 180 generallycooperates with the sending unit 140 to control operations of the mobileterminal 100 or execute data processing, a function or an operationassociated with an application program installed in the mobile terminalbased on the sensing signal. The sensing unit 140 may be implementedusing any of a variety of sensors, some of which will now be describedin more detail.

The proximity sensor 141 refers to a sensor to sense presence or absenceof an object approaching a surface, or an object located near a surface,by using an electromagnetic field, infrared rays, or the like without amechanical contact. The proximity sensor 141 may be arranged at an innerregion of the mobile terminal covered by the touch screen, or near thetouch screen.

When the touch screen is implemented as a capacitance type, theproximity sensor 141 can sense proximity of a pointer relative to thetouch screen by changes of an electromagnetic field, which is responsiveto an approach of an object with conductivity. When the touch screen isimplemented as a capacitance type, the proximity sensor 141 can senseproximity of a pointer relative to the touch screen by changes of anelectromagnetic field, which is responsive to an approach of an objectwith conductivity. In this case, the touch screen (touch sensor) mayalso be categorized as a proximity sensor.

The term “proximity touch” will often be referred to herein to denotethe scenario in which a pointer is positioned to be proximate to thetouch screen without contacting the touch screen. The term “contacttouch” will often be referred to herein to denote the scenario in whicha pointer makes physical contact with the touch screen. For the positioncorresponding to the proximity touch of the pointer relative to thetouch screen, such position will correspond to a position where thepointer is perpendicular to the touch screen. The proximity sensor 141may sense proximity touch, and proximity touch patterns (for example,distance, direction, speed, time, position, moving status, and thelike). In general, controller 180 processes data corresponding toproximity touches and proximity touch patterns sensed by the proximitysensor 141, and cause output of visual information on the touch screen.In addition, the controller 180 can control the mobile terminal 100 toexecute different operations or process different data (or information)according to whether a touch with respect to a point on the touch screenis either a proximity touch or a contact touch.

A touch sensor senses a touch (or a touch input) applied to the touchscreen (or the display unit 151) using any of a variety of touchmethods. Examples of such touch methods include a resistive type, acapacitive type, an infrared type, and a magnetic field type, amongothers.

As one example, the touch sensor may be configured to convert changes ofpressure applied to a specific part of the display unit 151, or convertcapacitance occurring at a specific part of the display unit 151, intoelectric input signals. The touch sensor may also be configured to sensenot only a touched position and a touched area, but also touch pressureand/or touch capacitance. A touch object is generally used to apply atouch input to the touch sensor. Examples of typical touch objectsinclude a finger, a touch pen, a stylus pen, a pointer, or the like.

When a touch input is sensed by a touch sensor, corresponding signalsmay be transmitted to a touch controller. The touch controller mayprocess the received signals, and then transmit corresponding data tothe controller 180. Accordingly, the controller 180 may sense whichregion of the display unit 151 has been touched. Here, the touchcontroller may be a component separate from the controller 180, thecontroller 180, and combinations thereof.

Meanwhile, the controller 180 may execute the same or different controlsaccording to a type of touch object that touches the touch screen or atouch key provided in addition to the touch screen. Whether to executethe same or different control according to the object which provides atouch input may be decided based on a current operating state of themobile terminal 100 or a currently executed application program, forexample.

The touch sensor and the proximity sensor may be implementedindividually, or in combination, to sense various types of touches. Suchtouches include a short (or tap) touch, a long touch, a multi-touch, adrag touch, a flick touch, a pinch-in touch, a pinch-out touch, a swipetouch, a hovering touch, and the like.

If desired, an ultrasonic sensor may be implemented to recognizelocation information relating to a touch object using ultrasonic waves.The controller 180, for example, may calculate a position of a wavegeneration source based on information sensed by an illumination sensorand a plurality of ultrasonic sensors. Since light is much faster thanultrasonic waves, the time for which the light reaches the opticalsensor is much shorter than the time for which the ultrasonic wavereaches the ultrasonic sensor. The position of the wave generationsource may be calculated using this fact. For instance, the position ofthe wave generation source may be calculated using the time differencefrom the time that the ultrasonic wave reaches the sensor based on thelight as a reference signal.

The camera 121, which has been depicted as a component of the input unit120, typically includes at least one a camera sensor (CCD, CMOS etc.), aphoto sensor (or image sensors), and a laser sensor.

Implementing the camera 121 with a laser sensor may allow detection of atouch of a physical object with respect to a 3D stereoscopic image. Thephoto sensor may be laminated on, or overlapped with, the displaydevice. The photo sensor may be configured to scan movement of thephysical object in proximity to the touch screen. In more detail, thephoto sensor may include photo diodes and transistors (TRs) at rows andcolumns to scan content received at the photo sensor using an electricalsignal which changes according to the quantity of applied light. Namely,the photo sensor may calculate the coordinates of the physical objectaccording to variation of light to thus obtain location information ofthe physical object.

The display unit 151 is generally configured to output informationprocessed in the mobile terminal 100. For example, the display unit 151may display execution screen information of an application programexecuting at the mobile terminal 100 or user interface (UI) and graphicuser interface (GUI) information in response to the execution screeninformation.

Also, the display unit 151 may be implemented as a stereoscopic displayunit for displaying stereoscopic images.

A typical stereoscopic display unit may employ a stereoscopic displayscheme such as a stereoscopic scheme (a glass scheme), anauto-stereoscopic scheme (glassless scheme), a projection scheme(holographic scheme), or the like.

The audio output module 152 may receive audio data from the wirelesscommunication unit 110 or output audio data stored in the memory 170during modes such as a signal reception mode, a call mode, a recordmode, a voice recognition mode, a broadcast reception mode, and thelike. The audio output module 152 can provide audible output related toa particular function (e.g., a call signal reception sound, a messagereception sound, etc.) performed by the mobile terminal 100. The audiooutput module 152 may also be implemented as a receiver, a speaker, abuzzer, or the like.

A haptic module 153 can be configured to generate various tactileeffects that a user feels, perceives, or otherwise experiences. Atypical example of a tactile effect generated by the haptic module 153is vibration. The strength, pattern and the like of the vibrationgenerated by the haptic module 153 can be controlled by user selectionor setting by the controller. For example, the haptic module 153 mayoutput different vibrations in a combining manner or a sequentialmanner.

Besides vibration, the haptic module 153 can generate various othertactile effects, including an effect by stimulation such as a pinarrangement vertically moving to contact skin, a spray force or suctionforce of air through a jet orifice or a suction opening, a touch to theskin, a contact of an electrode, electrostatic force, an effect byreproducing the sense of cold and warmth using an element that canabsorb or generate heat, and the like.

The haptic module 153 can also be implemented to allow the user to feela tactile effect through a muscle sensation such as the user's fingersor arm, as well as transferring the tactile effect through directcontact. Two or more haptic modules 153 may be provided according to theparticular configuration of the mobile terminal 100.

An optical output module 154 can output a signal for indicating an eventgeneration using light of a light source. Examples of events generatedin the mobile terminal 100 may include message reception, call signalreception, a missed call, an alarm, a schedule notice, an emailreception, information reception through an application, and the like.

A signal output by the optical output module 154 may be implemented insuch a manner that the mobile terminal emits monochromatic light orlight with a plurality of colors. The signal output may be terminated asthe mobile terminal senses that a user has checked the generated event,for example.

The interface unit 160 serves as an interface for external devices to beconnected with the mobile terminal 100. For example, the interface unit160 can receive data transmitted from an external device, receive powerto transfer to elements and components within the mobile terminal 100,or transmit internal data of the mobile terminal 100 to such externaldevice. The interface unit 160 may include wired or wireless headsetports, external power supply ports, wired or wireless data ports, memorycard ports, ports for connecting a device having an identificationmodule, audio input/output (I/O) ports, video I/O ports, earphone ports,or the like.

The identification module may be a chip that stores various informationfor authenticating authority of using the mobile terminal 100 and mayinclude a user identity module (UIM), a subscriber identity module(SIM), a universal subscriber identity module (USIM), and the like. Inaddition, the device having the identification module (also referred toherein as an “identifying device”) may take the form of a smart card.Accordingly, the identifying device can be connected with the terminal100 via the interface unit 160.

When the mobile terminal 100 is connected with an external cradle, theinterface unit 160 can serve as a passage to allow power from the cradleto be supplied to the mobile terminal 100 or may serve as a passage toallow various command signals input by the user from the cradle to betransferred to the mobile terminal therethrough. Various command signalsor power input from the cradle may operate as signals for recognizingthat the mobile terminal is properly mounted on the cradle.

The memory 170 can store programs to support operations of thecontroller 180 and store input/output data (for example, phonebook,messages, still images, videos, etc.). The memory 170 may store datarelated to various patterns of vibrations and audio which are output inresponse to touch inputs on the touch screen.

The memory 170 may include one or more types of storage mediumsincluding a flash memory type, a hard disk type, a solid state disk(SSD) type, a silicon disk drive (SDD) type, a multimedia card microtype, a card-type memory (e.g., SD or DX memory, etc.), a Random AccessMemory (RAM), a Static Random Access Memory (SRAM), a Read-Only Memory(ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM),a Programmable Read-Only memory (PROM), a magnetic memory, a magneticdisk, an optical disk, and the like. The mobile terminal 100 may also beoperated in relation to a network storage device that performs thestorage function of the memory 170 over a network, such as the Internet.

The controller 180 may typically control operations relating toapplication programs and the general operations of the mobile terminal100. For example, the controller 180 may set or release a lock state forrestricting a user from inputting a control command with respect toapplications when a status of the mobile terminal 100 meets a presetcondition.

The controller 180 can also perform the controlling and processingassociated with voice calls, data communications, video calls, and thelike, or perform pattern recognition processing to recognize ahandwriting input or a picture drawing input performed on the touchscreen as characters or images, respectively. In addition, thecontroller 180 can control one or a combination of those components inorder to implement various exemplary embodiments disclosed herein.

The power supply unit 190 receives external power or provides internalpower and supply the appropriate power required for operating respectiveelements and components included in the wearable device 100 under thecontrol of the controller 180. The power supply unit 190 may include abattery, which is typically rechargeable or be detachably coupled to theterminal body for charging.

The power supply unit 190 may include a connection port. The connectionport may be configured as one example of the interface unit 160 to whichan external charger for supplying power to recharge the battery iselectrically connected.

As another example, the power supply unit 190 may be configured torecharge the battery in a wireless manner without use of the connectionport. In this example, the power supply unit 190 may receive power,transferred from an external wireless power transmitter, using at leastone of an inductive coupling method which is based on magnetic inductionor a magnetic resonance coupling method which is based onelectromagnetic resonance.

Various embodiments described herein may be implemented in acomputer-readable medium, a machine-readable medium, or similar mediumusing, for example, software, hardware, or any combination thereof.

Referring to FIGS. 1B and 1C, the disclosed mobile terminal 100 includesa bar-like terminal body. However, the mobile terminal 100 mayalternatively be implemented in any of a variety of differentconfigurations. Examples of such configurations include watch type,clip-type, glasses-type, or a folder-type, flip-type, slide-type,swing-type, and swivel-type in which two and more bodies are combinedwith each other in a relatively movable manner, and combinationsthereof. Discussion herein will often relate to a particular type ofmobile terminal. However, such teachings with regard to a particulartype of mobile terminal will generally apply to other types of mobileterminals as well.

Here, considering the mobile terminal 100 as at least one assembly, theterminal body may be understood as a conception referring to theassembly.

The mobile terminal 100 will generally include a case (for example,frame, housing, cover, and the like) forming the appearance of theterminal. In this embodiment, the case is formed using a front case 101and a rear case 102. Various electronic components are interposed into aspace formed between the front case 101 and the rear case 102. At leastone middle case may be additionally positioned between the front case101 and the rear case 102.

The display unit 151 is shown located on the front side of the terminalbody to output information. As illustrated, a window 151 a of thedisplay unit 151 may be mounted to the front case 101 to form the frontsurface of the terminal body together with the front case 101.

In some embodiments, electronic components may also be mounted to therear case 102. Examples of such electronic components include adetachable battery 191, an identification module, a memory card, and thelike. In this case, a rear cover 103 is shown covering the electroniccomponents, and this cover may be detachably coupled to the rear case102. Therefore, when the rear cover 103 is detached from the rear case102, the electronic components mounted on the rear case 102 are exposedto the outside.

As illustrated, when the rear cover 103 is coupled to the rear case 102,a side surface of the rear case 102 may partially be exposed. In somecases, upon the coupling, the rear case 102 may also be completelyshielded by the rear cover 103. Meanwhile, the rear cover 103 mayinclude an opening for externally exposing a camera 121 b or an audiooutput module 152 b.

The cases 101, 102, 103 may be formed by injection-molding syntheticresin or may be formed of a metal, for example, stainless steel (STS),aluminum (Al), titanium (Ti), or the like.

As an alternative to the example in which the plurality of cases form aninner space for accommodating components, the mobile terminal 100 may beconfigured such that one case forms the inner space. In this case, amobile terminal 100 having a uni-body is formed in such a manner thatsynthetic resin or metal extends from a side surface to a rear surface.

Meanwhile, the mobile terminal 100 may include a waterproofing unit (notshown) for preventing introduction of water into the terminal body. Forexample, the waterproofing unit may include a waterproofing member whichis located between the window 151 a and the front case 101, between thefront case 101 and the rear case 102, or between the rear case 102 andthe rear cover 103, to hermetically seal an inner space when those casesare coupled.

The mobile terminal 100 may include a display unit 151, first and secondaudio output module 152 a and 152 b, a proximity sensor 141, anillumination sensor 142, an optical output module 154, first and secondcameras 121 a and 121 b, first and second manipulation units 123 a and123 b, a microphone 122, an interface unit 160, and the like.

Hereinafter, as illustrated in FIGS. 1B and 1C, description will begiven of the exemplary mobile terminal 100 in which the front surface ofthe terminal body is shown having the display unit 151, the first audiooutput module 152 a, the proximity sensor 141, the illumination sensor142, the optical output module 154, the first camera 121 a, and thefirst manipulation unit 123 a, the side surface of the terminal body isshown having the second manipulation unit 123 b, the microphone 122, andthe interface unit 160, and the rear surface of the terminal body isshown having the second audio output module 152 b and the second camera121 b.

However, those components may not be limited to the arrangement. Somecomponents may be omitted or rearranged or located on differentsurfaces. For example, the first manipulation unit 123 a may be locatedon another surface of the terminal body, and the second audio outputmodule 152 b may be located on the side surface of the terminal bodyother than the rear surface of the terminal body.

The display unit 151 is generally configured to output informationprocessed in the mobile terminal 100. For example, the display unit 151may display execution screen information of an application programexecuting at the mobile terminal 100 or user interface (UI) and graphicuser interface (GUI) information in response to the execution screeninformation.

The display module 151 may include at least one of a liquid crystaldisplay (LCD), a thin film transistor-LCD (TFT LCD), an organiclight-emitting diode (OLED), a flexible display, a three-dimensional(3D) display and an e-ink display.

The display unit 151 may be implemented using two display devices,according to the configuration type thereof. For instance, a pluralityof the display units 151 may be arranged on one side, either spacedapart from each other, or these devices may be integrated, or thesedevices may be arranged on different surfaces.

The display unit 151 may include a touch sensor that senses a touch withrespect to the display unit 151 so as to receive a control command in atouch manner. Accordingly, when a touch is applied to the display unit151, the touch sensor may sense the touch, and a controller 180 maygenerate a control command corresponding to the touch. Contents input inthe touch manner may be characters, numbers, instructions in variousmodes, or a menu item that can be specified.

On the other hand, the touch sensor may be configured in a form of afilm having a touch pattern and disposed between a window 151 a and adisplay (not illustrated) on a rear surface of the window, or may be ametal wire directly patterned on the rear surface of the window.Alternatively, the touch sensor may be formed integrally with thedisplay. For example, the touch sensor may be disposed on a substrate ofthe display, or may be provided inside the display.

In this way, the display unit 151 may form a touch screen together withthe touch sensor, and in this case, the touch screen may function as theuser input unit 123 (see FIG. 1A). In some cases, the touch screen mayreplace at least some of functions of a first manipulation unit 123 a.

The first audio output module 152 a may be implemented as a receiver fortransmitting a call sound to a user's ear and the second audio outputmodule 152 b may be implemented as a loud speaker for outputting variousalarm sounds or multimedia reproduction request sounds.

The window 151 a of the display unit 151 may include a sound hole foremitting sounds generated from the first audio output module 152 a.However, the present invention is not limited thereto, and the soundsmay be released along an assembly gap between the structural bodies (forexample, a gap between the window 151 a and the front case 101). In thiscase, a hole independently formed to output audio sounds may not be seenor may otherwise be hidden in terms of appearance, thereby furthersimplifying the appearance of the mobile terminal 100.

The optical output module 154 may be configured to output light forindicating an event generation. Examples of such events may include amessage reception, a call signal reception, a missed call, an alarm, aschedule alarm, an email reception, information reception through anapplication, and the like. When a user has checked a generated event,the controller 180 may control the optical output module 154 to stop thelight output.

The first camera 121 a may process image frames such as still or movingimages obtained by the image sensor in a capture mode or a video callmode. The processed image frames can then be displayed on the displayunit 151 or stored in the memory 170.

The first and second manipulation units 123 a and 123 b are examples ofthe user input unit 123, which may be manipulated by a user to provideinput to the mobile terminal 100. The first and second manipulationunits 123 a and 123 b may also be commonly referred to as a manipulatingportion. The first and second manipulation units 123 a and 123 b mayemploy any method if it is a tactile manner allowing the user to performmanipulation with a tactile feeling such as touch, push, scroll or thelike. The first and second manipulation units 123 a and 123 b may alsobe manipulated through a proximity touch, a hovering touch, and thelike, without a user's tactile feeling.

The drawings are illustrated on the basis that the first manipulationunit 123 a is a touch key, but the present disclosure may not benecessarily limited to this. For example, the first manipulation unit123 a may be configured with a mechanical key, or a combination of atouch key and a push key.

The content received by the first and second manipulation units 123 aand 123 b may be set in various ways. For example, the firstmanipulation unit 123 a may be used by the user to input a command suchas menu, home key, cancel, search, or the like, and the secondmanipulation unit 123 b may be used by the user to input a command, suchas controlling a volume level being output from the first or secondaudio output module 152 a or 152 b, switching into a touch recognitionmode of the display unit 151, or the like.

On the other hand, as another example of the user input unit 123, a rearinput unit (not shown) may be disposed on the rear surface of theterminal body. The rear input unit may be manipulated by a user to inputa command for controlling an operation of the mobile terminal 100. Thecontent input may be set in various ways. For example, the rear inputunit may be used by the user to input a command, such as power on/off,start, end, scroll or the like, controlling a volume level being outputfrom the first or second audio output module 152 a or 152 b, switchinginto a touch recognition mode of the display unit 151, or the like. Therear input unit may be implemented into a form allowing a touch input, apush input or a combination thereof.

The rear input unit may be disposed to overlap the display unit 151 ofthe front surface in a thickness direction of the terminal body. As oneexample, the rear input unit may be disposed on an upper end portion ofthe rear surface of the terminal body such that a user can easilymanipulate it using a forefinger when the user grabs the terminal bodywith one hand. However, the present disclosure may not be limited tothis, and the position of the rear input unit may be changeable.

When the rear input unit is disposed on the rear surface of the terminalbody, a new user interface may be implemented using the rear input unit.Also, the aforementioned touch screen or the rear input unit maysubstitute for at least part of functions of the first manipulation unit123 a located on the front surface of the terminal body. Accordingly,when the first manipulation unit 123 a is not disposed on the frontsurface of the terminal body, the display unit 151 may be implemented tohave a larger screen.

On the other hand, the mobile terminal 100 may include a finger scansensor which scans a user's fingerprint. The controller may usefingerprint information sensed by the finger scan sensor as anauthentication means. The finger scan sensor may be installed in thedisplay unit 151 or the user input unit 123.

The microphone 122 may be configured to receive the user's voice, othersounds, and the like. The microphone 122 may be provided at a pluralityof places, and configured to receive stereo sounds.

The interface unit 160 may serve as a path allowing the mobile terminal100 to interface with external devices. For example, the interface unit160 may be at least one of a connection terminal for connecting toanother device (for example, an earphone, an external speaker, or thelike), a port for near field communication (for example, an InfraredDaAssociation (IrDA) port, a Bluetooth port, a wireless LAN port, andthe like), or a power supply terminal for supplying power to the mobileterminal 100. The interface unit 160 may be implemented in the form of asocket for accommodating an external card, such as SubscriberIdentification Module (SIM), User Identity Module (UIM), or a memorycard for information storage.

The second camera 121 b may be further mounted to the rear surface ofthe terminal body. The second camera 121 b may have an image capturingdirection, which is substantially opposite to the direction of the firstcamera unit 121 a.

The second camera 121 b may include a plurality of lenses arranged alongat least one line. The plurality of lenses may be arranged in a matrixform. The cameras may be referred to as an ‘array camera.’ When thesecond camera 121 b is implemented as the array camera, images may becaptured in various manners using the plurality of lenses and imageswith better qualities may be obtained.

The flash 124 may be disposed adjacent to the second camera 121 b. Whenan image of a subject is captured with the camera 121 b, the flash 124may illuminate the subject.

The second audio output module 152 b may further be disposed on theterminal body. The second audio output module 152 b may implementstereophonic sound functions in conjunction with the first audio outputmodule 152 a, and may be also used for implementing a speaker phone modefor call communication.

At least one antenna for wireless communication may be disposed on theterminal body. The antenna may be embedded in the terminal body orformed in the case. For example, an antenna which configures a part ofthe broadcast receiving module 111 (see FIG. 1A) may be retractable intothe terminal body. Alternatively, an antenna may be formed in a form offilm to be attached onto an inner surface of the rear cover 103 or acase including a conductive material may serve as an antenna.

The terminal body is provided with a power supply unit 190 (see FIG. 1A)for supplying power to the mobile terminal 100. The power supply unit190 may include a batter 191 which is mounted in the terminal body ordetachably coupled to an outside of the terminal body.

The battery 191 may receive power via a power cable connected to theinterface unit 160. Also, the battery 191 may be (re)chargeable in awireless manner using a wireless charger. The wireless charging may beimplemented by magnetic induction or electromagnetic resonance.

On the other hand, the drawing illustrates that the rear cover 103 iscoupled to the rear case 102 for shielding the battery 191, so as toprevent separation of the battery 191 and protect the battery 191 froman external impact or foreign materials. When the battery 191 isdetachable from the terminal body, the rear case 103 may be detachablycoupled to the rear case 102.

An accessory for protecting an appearance or assisting or extending thefunctions of the mobile terminal 100 may further be provided on themobile terminal 100. As one example of the accessory, a cover or pouchfor covering or accommodating at least one surface of the mobileterminal 100 may be provided. The cover or pouch may cooperate with thedisplay unit 151 to extend the function of the mobile terminal 100.Another example of the accessory may be a touch pen for assisting orextending a touch input onto a touch screen.

The wireless communication unit 110, the input unit 120, and the outputunit 150 included in the mobile terminal 100 enables the mobile terminal100 to perform various types of communications with an external device.The input unit 120 receives communication information from a user. Thewireless communication unit 110 transmits information received throughthe input unit 120 to the external device and receives communicationinformation transmitted from the external device. The output unit 150outputs communication information received from the external device.

In this specification, communication information is defined as all kindsof information transmitted and received in order to performcommunications between a mobile terminal and an external terminal. Here,the communication information may have various types. For example, thecommunication information may be at least one of voice information,image information, and text information, but it is not limited thereto.The type of communication information may vary depending on acommunication method. Specifically, when the communication method is amethod using an auditory element, the communication information is voiceinformation, and when the communication method is a method using avisual element, the communication information may be at least one ofimage information and text information.

Meanwhile, in this specification, signals transmitted and receivedbetween telephone communication terminals are also defined ascommunication information.

Meanwhile, a different input element may be used for inputtingcommunication information according to its type. For example, audioinformation, video information, and text information may be input by themicrophone 122, the camera 121, and the touch screen 151, respectively.In more detail, the audio information may be input via the microphone122, the video information may be input through the camera 121, and thetext information may be input through the touch screen 151.

Also, a different output element may be used for outputtingcommunication information according to its type. For example, the audioinformation may be output via the speaker, and the video information andthe text information may be output through the touch screen 151.

As described above, the mobile terminal uses different input and outputelements depending on the communication method. Here, the communicationmethod is determined by an entity that generates a communication event.

The communication event described herein is defined as an event thattriggers communication using the mobile terminal, or as reception ofcommunication information itself.

When a communication event triggers communication, the communicationevent may be that receiving a call signal from an external device forcall connection. At this time, communication information may be receivedonly when there is a user request for the communication event. Forexample, in case of receiving the call signal from the external device,the mobile terminal may receive voice information only when there is acall connection request of the user.

As for the case of receiving communication information itself, acommunication event may be that receiving an image or a text messagefrom an external device.

The type of communication information is determined according to a typetransmitted by an external device that generated a communication event.

When a communication event is generated, the communication informationmay be stored in a preset memory. Here, the communication informationnot only includes information itself transmitted/received to and from anexternal terminal, but also means information that defines a specificcommunication event so that a specific communication event isdistinguished from another communication event. For example, thecommunication information may include a time at which a communicationevent is generated, a type of communication information, location ID ofcommunication information, information of an external terminal, and thelike. However, the communication information is not limited thereto.

The present disclosure is designed to make searching for thecommunication information easier.

To this end, the present disclosure stores the communication informationby matching it with another information. Hereinafter, a method forstoring the communication information and the communication event willbe described in detail.

FIG. 2 is a conceptual view illustrating a method in which information(data) is stored by matching communication information and communicationevent related information with another information.

Referring to FIG. 2, when a communication event 210 is generated in themobile terminal 100, the communication information is stored (220) in apreset memory. Here, communication information may be stored in adifferent database depending on a type of the communication event.

The controller 180 monitors (230, 240) content each time new informationis stored in the memory, and matches the new information with otherinformation that can be collected in the mobile terminal 100 beforestoring it. Specifically, the controller 180 collects current locationinformation of the mobile terminal 100 through GPS embedded in themobile terminal 100 each time a communication event is generated (see260). In addition, the controller 180 extracts location information fromcommunication information. The collected location information andextracted location information (hereinafter referred to as “locationinformation”) are matched with at least one piece of information relatedto the communication information before being stored. Here, the locationinformation is stored in a separate database 250. That is, the locationinformation is not necessarily stored in the same database as thecommunication information. In this specification, the database in whichthe location information is stored is referred to as a locationinformation database.

The controller 180 may store the location information and location ID ofcommunication information together. Accordingly, communicationinformation matched with the location information can be reached(accessed) through the location information.

Meanwhile, the controller 180 firstly matches location information ofthe mobile terminal, which is collected when a communication event isgenerated, with communication information. The controller 180 may matchextracted location information with the communication information onlywhen location related information is extracted from the communicationinformation. When the location related information is extracted from thecommunication information, a plurality of pieces of location informationmay be matched to the communication information.

Meanwhile, the controller 180 may compare collected location informationwith pre-stored location information each time location information ofthe mobile terminal 100 is collected in response to an occurrence of acommunication event. At this time, when collected location informationsatisfies a preset condition, the controller 180 may assign arepresentative name to the collected location information, and thenstore it by matching the name with the collected location information.

The preset condition will be described in detail. The controller 180compares the collected location information with pre-stored locationinformation. When location information the same as the collectedlocation information is found in the pre-stored location information,the controller 180 determines whether there is a representative namematched to the location information found. When there is a matchedrepresentative name, the controller 180 matches the representative nameto the collected location information to store it.

When there is no matched representative name, and location informationthe same as the collected location information is searched for more thana predetermined number of times, the controller 180 may create arepresentative name based on a time period at which the informationlocation is searched. For example, the time period for searching thelocation information for the predetermined number of times or more ismainly collected in the evening and morning hours, the controller 180may set a representative name as “home” for the location information. Inthis case, the same representative name is matched to all the locationinformation retrieved for the predetermined number of times or more.

Meanwhile, it is not limited to the communication event, the controller180 may also store location information by matching event relatedinformation with location information each time a new event is generatedin the mobile terminal 100.

Here, the event may include application execution, photographing,storing new data, etc. For example, the controller 180 may activate thecamera 121 in response to a user request to capture an image, and storethe captured image. When storing the captured image, the controller 180may match location information to the captured image.

As another example, the controller 180 may receive personal scheduleinformation input from the user and store it in the memory 170. At thistime, the controller 180 may store the personal schedule information bymatching location information to it.

As described above, the mobile terminal according to the presentdisclosure stores information (data) by matching location information tocommunication information each time a communication event is generated,and stores information (data) by matching location information to theevent related information for storing each time a new event isgenerated.

Thereafter, when a request for communication information searching isinput from the user, the controller 180 retrieves pre-storedcommunication information using a search query (search term) input bythe user.

Hereinafter, an exemplary embodiment of searching for communicationinformation and event related information in response to a searchrequest of a user will be described in detail.

FIG. 3 is a block diagram illustrating searching for communicationinformation and event related information in a mobile terminal accordingto one embodiment of the present disclosure.

Referring to FIG. 3, a plurality of applications 310 may be installed inthe mobile terminal. An event or a communication event may be generatedthrough the applications.

For example, the mobile terminal according to the present disclosure maysend and receive text messages with another terminal through a messengerapplication. Here, sent and received text messages are stored.

As another example, the mobile terminal according to the presentdisclosure may activate the camera 121 and store an image received bythe camera 121 through an image capture application.

As described above, applications are executed and information ofapplication execution results are stored in the mobile terminal. Also,as aforementioned, information of the application execution result maybe stored by matching location information to it.

Meanwhile, an application 320 for a search query input to retrievecommunication information and event related information may be installedin the mobile terminal. In FIG. 3, an application for receiving a searchquery given in spoken form is only disclosed. However, besides the voiceinput, the user may search for communication information and eventrelated information through a text input, an image input, and the like.

When a search query is input via the application 320, the controller 180transmits the search query to a separate API 330, and extracts at leastone keyword 340 from the search query. Thereafter, the controller 180searches a database 350 for information corresponding to the keyword tooutput a search result. The controller 180 searches not only thedatabase 350 shown in FIG. 3 but also the above-described locationinformation database for communication information and event relatedinformation.

First, a description will be given of an embodiment in which thecontroller 180 only retrieves information corresponding to a keywordfrom the database 350 illustrated in FIG. 3.

FIG. 4 is a flowchart illustrating one embodiment of searchingpre-stored data using time information included in a search query, FIGS.5 and 6 are conceptual views illustrating searching pre-stored datausing location information or time information included in a searchquery, in accordance with one embodiment of the present disclosure.

Referring to FIG. 6, when the user inputs a search query (S110), thecontroller 180 checks if a keyword indicating time information exists inthe search query (S120). When a keyword related to time informationexists, the controller 180 extracts the keyword and then retrievesinformation corresponding to the keyword from the database 350 to outputa search result.

Here, the keyword related to time information may be a date, a time,etc. representing an absolute time point, or a term representing arelative time point. For example, the time-related keyword may be timeinformation based on the current time point such as “yesterday”, “lastweek”, and the like.

When the keyword related to time information represents the relativetime point, the controller 180 may change the time information of thekeyword to another time information based on the current time.

In one embodiment, referring to FIG. 4, when a search query “Courierservice Magok-dong last week” is input (410) from the user, thecontroller 180 extracts a time-related keyword “last week” from thesearch query. The controller 180 changes (420) the extracted keyword toanother time information based on the current time. Thereafter, thecontroller 180 searches the database 350 for information including theconverted time information.

Meanwhile, the controller 180 searches for information including anotherkeyword “courier service” included in the search query within a searchresult. Then the controller 180 outputs (470) the final search result.

Referring to FIG. 5, the user may input a search query to retrieve alist of calls sent or received with another terminal. In one embodiment,when the user inputs “Mom last year”, the controller 180 extracts akeyword “last year” related to time information from the search query,and extracts the remaining keyword “Mom”. Thereafter, the controller 180changes “last year” to another time information based on the currenttime. The controller 180 searches the database 350 for informationcorresponding to the converted time information. Then the controller 180searches for information including “Mom” within a search result tooutput a final search result.

As described above, the present disclosure allows the user to retrieveinformation desired from the database by combining a time-relatedkeyword with another keyword included in a search query.

Hereinafter, an embodiment in which a user searches for informationdesired using a location-related keyword included in a search query inthe mobile terminal according to the present disclosure will bedescribed.

FIG. 7 is a conceptual view illustrating one embodiment of extracting alocation-related keyword included in a search query in the mobileterminal according to the present disclosure.

The controller 180 extracts a keyword associated with locationinformation from a search query input by the user. Here, the keyword maybe an address, a coordinate, a place name, a building name, and thelike. However, the keyword is not limited thereto. In one exemplaryembodiment, the controller 180 may compare the keyword with a pluralityof pre-stored representative names to extract a keyword from the searchquery. For example, when the word of “home” is included in the searchquery, the controller 180 extracts the “home” as a keyword.

The controller 180 extracts a keyword related to location informationfrom the search query, and then retrieves location information identicalto the extracted keyword from the location information database.

Here, the controller 180 may retrieve location information identical tothe keyword from the location information database. The controller 180may also change the keyword initially to search for location informationthe same as the changed data. For example, when the extracted keyword isone of the plurality of pre-stored representative names, the controller180 may change the representative name to any corresponding one of anaddress, a coordinate, a place name, and a building name to search ifthe changed data exists in the location information database.

Meanwhile, when a plurality of location-related keywords is included inthe search query, the controller 180 may use part of the plurality ofkeywords for searching the database. In one embodiment, when onelocation-related keyword in a search query is a sub-concept of anotherlocation-related keyword, the controller 180 may only use the anotherlocation-related keyword for searching the database.

For example, referring to FIG. 7, when “Cat Magok-dong Gangseo-gu Seoul”is input (510) as a search query, the controller 180 may extractkeywords “Magok-dong”, “Gangseo-gu”, and “Seoul” (550) related tolocation information from the search query. In this process, variousalgorithms for extracting the keywords may be used. For instance, theAho-Corasick algorithm 540 may be used for keyword extraction.

Here, since the “Gangseo-gu”, a place of address, is a subclass of“Seoul”, and the “Magok-dong”, a place of address, is a subclass of“Gangseo-gu” and “Seoul”, the controller 180 searches if locationinformation corresponding to “Magok-dong” exists in the locationinformation database.

Referring back to FIG. 4, the controller 180 extracts a time-relatedkeyword, a location-related keyword, and the remaining keyword from thesearch query, and combines all the extracted keywords to search forinformation desired by the user.

Referring to FIG. 4, when a search query of “Courier service Magok-donglast week” is input (410) by the user, the controller 180 extracts akeyword “last week” related to time information from the search query.The controller 180 changes (420) the extracted keyword to another timeinformation based on the current time.

Meanwhile, the controller 180 extracts “Magok-dong” from the searchquery, and then searches for location information including“Magok-dong”. Then the controller 180 retrieves (450) data that includeslocation ID stored together with the searched location information andthe converted time information from the database 350.

Thereafter, the controller 180 retrieves (460) data that includes“courier service” among the data including location ID stored togetherwith the searched location information and the converted timeinformation.

Lastly, the controller 180 outputs (470) a final search data. Accordingto this present disclosure, information desired by the user can beaccurately retrieved by combining various kinds of keywords included ina search query.

Referring back to FIG. 5, the user may input a search query to retrievea list of calls sent or received with another terminal. In oneembodiment, when the user inputs “Jeju Island Mom” for the search query,the controller 180 extracts a keyword “Jeju Island” related to locationinformation and extracts the remaining keyword “Mom” from the searchquery. Then the controller 180 retrieves location information including“Jeju Island” from the location information. Thereafter, the controller180 retrieves data that includes location ID stored together with thesearched location information and “Mom” from the database 350, andoutputs a search result.

Meanwhile, the above-described searching method is not limited tocommunication information, and it may be used for all event-relatedinformation.

FIG. 8 is a conceptual view illustrating one embodiment of retrieving animage previously captured by using time and location information.

Specifically, the aforementioned searching method may be used forretrieving an image previously captured and stored. When storing animage captured through the camera 121 of the mobile terminal 100 in thedatabase 350, the controller 180 stores the captured image together witha time and a place taken. The controller 180 may utilize the time andthe place stored together with the captured image for searching.

For example, referring to FIG. 8, when a search query of “Jeju Islandlast year” is input into a gallery application, the controller 180extracts a time-related keyword “last year” and a location-relatedkeyword “Jeju Island” from the search query.

Thereafter, the controller 180 changes the extracted keyword “last year”to another time information based on the current time. Meanwhile, thecontroller 180 retrieves location information including “Jeju Island”from the location information database. Then the controller 180 searchesthe database 350 for data that includes location ID stored together withthe retrieved location information and the converted time information.After that, the controller 180 outputs the searched data.

Meanwhile, the controller 180 stores the captured image together withcategory data defining a type of the image. Here, the category data maybe generated by a predetermined algorithm. If it takes a long time tocreate the category data, the category data does not necessarily have tobe generated immediately after image capturing. In one embodiment, thecategory data may be stored after being collectively generated at apredetermined time.

As described above, according to the present disclosure, when a specificevent is generated in the mobile terminal, information related to theevent and additional information are stored together to make it easy tosearch later.

Meanwhile, according to the present disclosure, specific information isprovided to the user by retrieving the specific information based on aterminal usage pattern of the user even when there is no search requestof the user.

FIG. 9 is a conceptual view illustrating one embodiment in whichspecific information is provided to a user by retrieving the specificinformation based on a terminal usage pattern of the user even whenthere is no search request of the user.

Specifically, when a preset condition is satisfied, the controller 180retrieves information from the database 350 and outputs a search result.Here, the preset condition may be that executing a specific application.In this case, a kind of database searched by the controller 180 may varydepending on a type of application executed.

In one embodiment, the controller 180 may search the database 350 formost called contacts each time the user executes a call application.

Meanwhile, when the preset condition is determined to be satisfied, thecontroller 180 searches the database 350 by using at least one of timeinformation and location information. Specifically, in this case, thecontroller 180 may use the current time from a time point when thepreset condition is satisfied for searching, or may collect the currentlocation from the GPS to use for searching when the preset condition issatisfied.

For example, when the user executes the call application, the controller180 may search for a list of calls (call history) including timeinformation within a predetermined time period (or range) based on thecurrent time. Then the controller 180 may display corresponding contactsfrom the most retrieved contact number. According to the presentdisclosure, most frequently contacted numbers at a specific time(period) can be provided to the user.

Further, when there is a rejected call within a specific time pointbased on the time at which the call application is executed, thecontroller 180 may display the rejected call as a recommended contactnumber.

As another example, when the user executes the call application, thecontroller 180 receives the current location information of the mobileterminal 100 through the GPS, and uses the location information toretrieve location information identical to it from the locationinformation database. Thereafter, the controller 180 may output a listof calls matched to the searched location information by using locationID stored together with it. Further, the controller 180 may displaycorresponding contacts from the most frequently contacted number amongthe list of calls matched to the searched location information.According to the present disclosure, when the mobile terminal 100 islocated in a specific place, most frequently contacted numbers in thespecific place can be provided.

Referring to FIG. 9, when the user executes the call application, arecommended contact number may be displayed in one area 610 of anexecution screen of the call application. When the user applies a touchto the one area 610, all recommended contacts 620 searched by thecontroller 180 may be displayed. Meanwhile, when the user touches theone area 610, a list of received and sent calls with the recommendedcontacts may be displayed as illustrated in (c) of FIG. 9.

Meanwhile, the present disclosure periodically retrieves pre-storedcommunication information for providing a reminder to the user.

FIG. 10 is a flowchart illustrating one embodiment of providing areminder to a user using a pre-stored text message, and FIGS. 11A and11B are conceptual views illustrating providing a reminder to a userusing a pre-stored text message, in accordance with one embodiment ofthe present disclosure.

Referring to FIG. 10, when a text message is received from an externalterminal (S210), the controller 180 stores the received text message ina preset database. Accordingly, a plurality of text messages is storedin the preset database.

Thereafter, the controller 180 collects the received text message(S230), and extracts a keyword. The controller 180 determines whetherthe corresponding text message needs to be reminded using the extractedkeyword (S240).

The present disclosure extracts a specific keyword from a text messagestored in the database 350, and determines a type of the stored textmessage using the specific keyword.

For example, the controller 180 extracts a specific keyword from thetext message to determine its type. The text message may be a couponwith an expiration date or a booking confirmation. The controller 180may extract a keyword related to a coupon or a reservation included inthe text message to analyze. In one embodiment, the controller 180extracts keywords such as “reservation confirmed”, “reservation number”,“reservation date”, and the like included in the text message to analyzewhether the text message needs to be reminded.

The controller 180 stores a text message to be reminded in a separatedatabase (S250). The controller 180 stores a due date for reminderstogether with the text message to be reminded.

Thereafter, the controller 180 checks if there is a text message forreminders at a designated time every day (S260). When a text message forreminders exists, the controller 180 outputs a reminder message.

Meanwhile, when a date is changed (S310) while the reminder message isbeing displayed, the controller 180 stops displaying the remindermessage (S320), and analyzes the pre-stored database to output a newreminder message.

Referring to FIG. 11A, the controller 180 extracts a keyword“reservation confirmed” from a text message to determine that the textmessage is a booking confirmation message. Thereafter, the controller180 extracts time information from the text message and outputs areminder message before the time corresponding to the extracted timeinformation arrives.

Referring to FIG. 11B, the controller 180 extracts a keyword “couponnumber” from a text message to determine that the text message is a textmessage coupon. Thereafter, the controller 180 extracts time informationfrom the text message and outputs a reminder message before the timecorresponding to the extracted time information arrives.

As described above, the present disclosure analyzes a received textmessage to provide reminders to the user.

Hereinafter, a method for inputting a search query for a database searchwill be described. The search query may be input via an artificialintelligence (AI) agent provided in the mobile terminal. Hereinafter,the AI agent will be described in detail.

FIGS. 12A and 12B are conceptual views illustrating a state in which twoor more AI agents are interconnected.

An agent is an autonomous process or entity that acts on behalf of auser for achieving a specific goal. Here, the agent is not a conceptexisting independently of another, but it may be understood as asoftware installed in the mobile terminal. However, the presentdisclosure is not limited thereto, the agent may be a system including aserver capable of performing wireless communications with the mobileterminal, and the like. The agent interacts with the user and anexternal environment through the components included in the mobileterminal 100.

Specifically, the agent receives command information from the userthrough the components included in the mobile terminal, and determines aspecific control to perform based on the command information.Thereafter, the agent receives information needed to perform thespecific control through the components included in the mobile terminal.

In this specification, information received from the user is defined ascommand information to specify a control to be performed by the agent.The command information may be input via various elements, and may be adifferent type depending on the input element. For example, the commandinformation may be input by using at least one of a voice, video, andtext element.

The agent may be classified into a voice (or speech) recognition agent,a text recognition agent, and the like according to a command inputelement.

For example, the voice recognition agent receives a spoken command(information) from the user through the microphone 122 included in themobile terminal 100, and performs a control related to the receivedspoken command.

As another example, the text recognition agent receives a textualcommand (information) from the user through the touch screen 151, andperforms a control related to the received text command.

Meanwhile, the command information may include various kinds ofinformation. For example, the command information may include a type offunctions operable in the mobile terminal, start/end time of functionexecution, start/end criteria for function execution, and the like.

Meanwhile, the AI agent described herein analyzes the meaning of commandinformation by using data stored in the controller 180 and theartificial intelligence unit 130. Specifically, when the commandinformation is a command of natural language, the AI agent uses anatural language processing function of the controller 180 to analyzewhat a user's intended task is. In more detail, the AI agent extractsthe type of function that may be operated in the mobile terminal,start/end time of function execution, start/end criteria for functionexecution, and the like based on the command information of the naturallanguage, and perform control accordingly.

In addition, the AI agent predicts (or infers) at least one operation ofthe mobile terminal from the command information using informationlearned by the aforementioned machine learning technology, and controlsthe mobile terminal such that the most feasible operation among thepredicted operations is performed.

In this specification, the AI agent is described to control thecomponents of the mobile terminal 100 for convenience of explanation.However, the AI agent is substantially implemented by data stored in thecontroller 180 and the artificial intelligence unit 130, so it may beunderstood that the controller 180 controls the components of the mobileterminal 100 by using the data stored in the artificial intelligenceunit 130. In other words, the control by the AI agent and the control bythe controller 180 using the data stored in the artificial intelligenceunit 130 are substantially the same in this specification.

Meanwhile, the AI agent according to the present disclosure may beapplied to various types of agents. For example, the AI agent accordingto the present disclosure may be applied to a voice recognition agentand a text recognition agent, respectively.

As described in FIGS. 1 to 9, an entity that performs a search forspecified criteria in pre-stored database and outputs a search resultwhen a search query is input by a user or a preset condition issatisfied.

The mobile terminal 100 may include a plurality of different AI agents.Part of the plurality of AI agents may be installed at the time ofshipment, and another part of the remaining AI agents may be installedby a user.

Reception of a search query from the user and a search using the searchquery may be performed by different agents. Hereinafter, a method forproviding a search result corresponding to a search query input from theuser by interconnecting different agents will be described.

First, the method of inputting a search query through an AI agent willbe described. Although the AI agent described below is a voicerecognition AI agent, the present disclosure is not limited thereto.

The user may execute the voice recognition AI agent by speaking aspecific keyword. For example, when the user speaks a wake-up signal (orwake-up word), the AI agent is executed.

Here, the mobile terminal may always keep the microphone 122 on in orderto receive the wake-up signal from the user. However, the mobileterminal does not perform control related to voice information receiveduntil the wake-up signal is received.

When the wake-up signal is received, the mobile terminal executes thevoice recognition AI agent, and displays an execution screen of thevoice recognition AI agent instead of the currently displayed screeninformation. The currently displayed screen information may be a homescreen of the mobile terminal or an application execution screen.Meanwhile, a graphic object for informing whether or not activate voicerecognition may be displayed on the execution screen of the voicerecognition AI agent.

Meanwhile, when the wake-up signal is received, the mobile terminal maydisplay an execution screen of the voice recognition AI agent, andactivate voice recognition. In more detail, when the wake-up signal isreceived, the mobile terminal may execute the voice recognition AI agentand immediately activates the voice recognition. At this time, a graphicobject for informing whether or not activate the voice recognition and atext indicating that the voice recognition is activated may bedisplayed.

The mobile terminal may include a plurality of voice recognition AIagents. A different AI agent may be activated according to a type ofwake-up signal received through the microphone 122. Thereafter, a searchcommand input from the user is performed by an activated AI agent(hereinafter referred to as “first AI agent”).

When a search command from the user is received, the voice recognitionAI agent converts the spoken command into written text, and extracts asearch keyword from the converted text. Thereafter, the extracted searchkeyword is transmitted to a search AI agent (hereinafter referred to as“second AI agent”).

Here, a transmission path (or channel) of the search keyword may varydepending on a type of the voice recognition AI agent. For example,referring to FIG. 12A, a first AI agent 710 a extracts a search keywordfrom a search command of the user, and immediately transmits theextracted search keyword to a second AI agent 720. Then the second AIagent 720 performs a search using the method described in FIGS. 2 to 9to transfer a search result to the first AI agent 710 a. The first AIagent 710 a outputs the received search result. That is, the searchresult may be displayed on an execution screen of the first AI agent 710a.

On the other hand, the first AI agent may not by interconnected with thesecond AI agent depending on a type of the AI agent. For example, if afirst AI agent is installed at the time of mobile terminal shipment, anda second AI agent is installed by the user, they may not beinterconnected with each other. In this case, referring to FIG. 12B, thefirst AI agent 710 b transmits a search keyword extracted from a searchcommand of the user to a specific application 730 that can beinterconnected with the second AI agent 720. The specific application730 transmits the extracted search keyword to the second AI agent 720.Thereafter, the second AI agent 720 transmits a search result to thespecific application 730, and the specific application 730 transmits thesearch result to the first AI agent 710 b.

The method illustrated in FIG. 12B may be utilized when interconnectingdifferent AI agents installed in respective terminals.

FIG. 13 is a conceptual view illustrating one embodiment in which asearch command is input via an AI agent installed in an image displaydevice, and a search is performed in a mobile terminal. FIG. 14 is aconceptual view illustrating one embodiment in which a search command isinput via an AI agent installed in a mobile terminal, and a searchresult is displayed on an image display device.

Referring to FIG. 13, the user may input a search command through animage display device. The AI agent may be installed in the image displaydevice, and the AI agent installed in the image display device receivesa spoken search command from the user, converts the spoken command intowritten text, and extracts a search keyword. The extracted searchkeyword is transmitted to a specific application of the mobile terminal,and the specific application transfers the search keyword to a search AIagent installed in the mobile terminal. The search AI agent performs asearch using the search keyword and transmits a search result to thespecific application. Thereafter, the specific application transfers thesearch result to the image display device. Accordingly, the searchresult in the mobile terminal can be displayed on the image displaydevice.

Referring to FIG. 14, the user may input a search command through themobile terminal. An AI agent installed in the mobile terminal receives aspoken search command from the user, converts the spoken search commandinto written text and extracts a search keyword. The extracted searchkeyword is transmitted to a specific application of the mobile terminal,and the specific application transfers the search keyword to a search AIagent installed in the mobile terminal. The search AI agent performs asearch using the search keyword and transmits a search result to thespecific application. Then the specific application transfers the searchresult to an image display device. Accordingly, the search result in themobile terminal can be displayed on the image display device.

As described above, in the present disclosure, different agents can beinterconnected to each other by using a specific application as a mediumwhen directly connecting the different agents are not available.

The present disclosure can be implemented as computer-readable codes ina program-recorded medium. The computer-readable medium may include alltypes of recording devices each storing data readable by a computersystem. Examples of such computer-readable media may include hard diskdrive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM,CD-ROM, magnetic tape, floppy disk, optical data storage element and thelike. Also, the computer-readable medium may also be implemented as aformat of carrier wave (e.g., transmission via an Internet). Thecomputer may include the controller 180 of the terminal. The abovedetailed description should not be limitedly construed in all aspectsand should be considered as illustrative. Therefore, all changes andmodifications that fall within the metes and bounds of the claims, orequivalents of such metes and bounds are therefore intended to beembraced by the appended claims.

1-15. (canceled)
 16. A mobile terminal, comprising: a GPS configured toreceive location information of the mobile terminal; a wirelesscommunication unit configured to perform wireless communication with anexternal terminal; and a controller configured to: in response to acommunication event between the external terminal and the mobileterminal, store the location information received from the GPS withcommunication information about the communication event in a database,and in response to an input search query having a location, output aresult of communication events in which the location included in theinput search query matches the location information stored with thecommunication information.
 17. The mobile terminal of claim 16, whereinthe controller is further configured to: extract a keyword related tothe location included in the search query, compare the extracted keywordwith the location information stored with the communication information,and output the at least one piece of information corresponding to thesearch query.
 18. The mobile terminal of claim 17, wherein thecontroller is further configured to: assign a representative name to thelocation information stored with the communication information.
 19. Themobile terminal of claim 18, wherein the controller is furtherconfigured to: change the extracted keyword to location informationcorresponding to any one representative name of pre-storedrepresentative names, compare the changed location information with thelocation information stored with the communication information, andoutput the at least one piece of information corresponding to the searchquery.
 20. The mobile terminal of claim 18, wherein the controller isfurther configured to: when same location information is collected morethan a predetermined number of times within a predetermined time period,assign the representative name to the location information stored withthe communication information, and store the location information bymatching the location information with the representative name.
 21. Themobile terminal of claim 16, wherein the controller is furtherconfigured to: extract a keyword related to the location and a keywordrelated to time information included in the search query, and outputcommunication events corresponding to each of the extracted keywordsfrom the location information stored with the communication information.22. The mobile terminal of claim 21, wherein the controller is furtherconfigured to: output communication events corresponding to theextracted keyword related to the time information.
 23. The mobileterminal of claim 16, wherein the controller is further configured to:determine that the search query is input when a preset condition issatisfied, and use the location information stored with thecommunication information and the search query to output the at leastone piece of information corresponding to the search query.
 24. Themobile terminal of claim 23, wherein the preset condition is a specificapplication executing.
 25. The mobile terminal of claim 23, wherein thecontroller is further configured to: collect location information fromthe GPS, compare the collected location information with the locationinformation stored with the communication information, and output the atleast one piece of information corresponding to the collected locationinformation, when the preset condition is satisfied.
 26. The mobileterminal of claim 25, wherein the controller is further configured to:collect current time information and output the at least one piece ofinformation corresponding to the collected time information from thelocation information stored with the communication information when thepreset condition is satisfied.
 27. A method of controlling a mobileterminal, the method comprising: receiving, via a GPS of the mobileterminal, location information of the mobile terminal; performingwireless communication, via a wireless communication unit of the mobileterminal, with an external terminal; in response to a communicationevent between the external terminal and the mobile terminal, storing,via a controller of the mobile terminal, the location informationreceived from the GPS with communication information about thecommunication event in a database; and in response to an input searchquery having a location, outputting, via the controller, a result ofcommunication events in which the location included in the input searchquery matches the location information stored with the communicationinformation.
 28. The method of claim 27, further comprising: extractinga keyword related to the location included in the search query;comparing the extracted keyword with the location information storedwith the communication information; and outputting the at least onepiece of information corresponding to the search query.
 29. The methodof claim 28, further comprising: assigning a representative name to thelocation information stored with the communication information.
 30. Themethod of claim 29, further comprising: changing the extracted keywordto location information corresponding to any one representative name ofpre-stored representative names; comparing the changed locationinformation with the location information stored with the communicationinformation; and outputting the at least one piece of informationcorresponding to the search query.
 31. The method of claim 29, furthercomprising: when same location information is collected more than apredetermined number of times within a predetermined time period,assigning the representative name to the location information storedwith the communication information; and storing the location informationby matching the location information with the representative name. 32.The method of claim 27, further comprising: extracting a keyword relatedto the location and a keyword related to time information included inthe search query; and outputting communication events corresponding toeach of the extracted keywords from the location information stored withthe communication information.
 33. The method of claim 32, furthercomprising: outputting communication events corresponding to theextracted keyword related to the time information.
 34. The method ofclaim 27, further comprising: determining that the search query is inputwhen a preset condition is satisfied, and use the location informationstored with the communication information and the search query to outputthe at least one piece of information corresponding to the search query.35. The method of claim 34, wherein the preset condition is a specificapplication executing.
 36. The method of claim 34, further comprising:collecting location information from the GPS; comparing the collectedlocation information with the location information stored with thecommunication information; and outputting the at least one piece ofinformation corresponding to the collected location information, whenthe preset condition is satisfied.
 37. The method of claim 36, furthercomprising: collecting current time information and outputting the atleast one piece of information corresponding to the collected timeinformation from the location information stored with the communicationinformation when the preset condition is satisfied.